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Agenda 


•  NAVAIR 

•  Team  Process  Integration  (TPI) 

•  Team  "X"  Process  (TxP) 

•  Time-Based  Postmortem 

•  Size- Based  Postmortem 

•  Quality-Based  Postmortem 
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What  is  NAVAIR? 


•  NAVAI R  is  the  Naval  Air  Systems  Command 

•  Develop,  acquire,  and  support  the  aircraft  and 
related  weapons  systems  used  by  U.S.  Navy  and 
Marine  Corps 

•  Our  goal  Is  to  provide  the  fleet  with  quality 
products  that  are  both  affordable  and  available 
when  most  needed 

•  Our  support  extends  across  the  entire  life  span  of  a 
product,  including  all  upgrades  and  modifications 

to  that  product 
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Team  Process  Integration  (TPI) 
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Models  and  Processes 


Capability  Maturity  Models:  m  w  «i  «i 

Reference  for  organizations  •  igi  ■■■  ™*  •  \ 

building  process  capability  •  I  *»■**»■»  *'■'*'■**  | '  • 

\m  \m  \m  m  w/  \i/  m  n/ 


Team  Processes: 

Processes  for  teams 
building  quality  products 
on  cost  and  schedule 
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Personal  Processes: 

Processes  used  to  train 
individual  skill  and 
discipline 
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Key  Team  Process  Framework 


Goals,  products  &  services, 
top-down  &  bottom-up 
planning  with  load 
balancing,  risk  assessment 


Develop  products 
&  services 


(  Productivity  \ 

V  database  ) 

(  Process  \ 

V  Phases  )  ^ 

(  Resources 
available 


•Track  and  report 
progress  periodically 

•Update  historical 
data  used  for  future 
planning 
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Team  Measures  and  Metrics 


•  Each  team  member  gathers 
four  basic  measures 

-  Times 

-  Sizes 

-  Mistakes 

-  Task  completion  dates 


Charts  and  tables  of  project 
metrics  are  available 
(updated  in  real  time) 


Direct  Hours 


Earned  Value 


^JgJxJ 


Tasks  in  Progress 


many  more... 
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NAVAI R  TPI 


•  Success  of  software  teams  using  TSP  led  their 
organizations  to  ask  for  same  performance  on 
other  teams 

-  Worked  with  the  SEI  to  develop  approach 

-  Based  on  same  TSP  fundamental  principles 


•  NAVAI  R  approach  has  become  TPI  for  all  teams 

-  Teams  plan  all  work  from  first  launch  forward 

-  Work  is  based  on  all  products  and  services  defined  in 
process  modeling 

-  PSP  for  Engineers  training  planned  as  part  of  project  if 
appropriate 


Slide  11 


Approved  for  Public  Release 


nav^Tai  r 


J  ust-in-TimeTPI  Training 

Learning.  Doing. 
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Team  [topic-name]  Process 

(TxP) 
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lip  TPI  Pluses  &  Minuses 

+  A  detailed  plan! 

+  Ability  to  track  progress  (weekly) 

+  Improved  estimating  (over  cycles) 

-No  mature  processes 

-"Where  do  we  put  mistake-fixing  phases?" 

-  No  defect  type  standards 

-"What  kinds  of  mistakes  do  I  make?" 

-  No  quality  planning 

-"Will  our  plan  produce  a  good  product?" 
-No  qualityjndjcators  (e^g.,  A/FR) 
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CMMI.  TSCHWH Relationship 


TRP 
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Us  TPI  is  Only  a  Waypoint 

•  TPI  teams  will  hit  a  glass  ceiling 

•  TPI  teams  need  to  evolve  to  achieve  TSP- 
like  performance  (become  a  TxP  team) 

•  What  else  does  a  TPI  team  have  to  do  in 
order  to  become  a  TxP  team? 

•  What  does  a  TSP  team  do? 
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What  Does  a  TSP  Team  Do? 


(Plan  the  Work) 


(Work  the  Plan) 


(Analyze  the  Data) 


Launch 


Weekly  Meetings  and  Day- to- Day  Actions 


Tvoical  TSP  Cycle 


Postmortem 
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TxP  Planning  Activities 


Project  and  Management  Objectives 
Team  Goals  and  Roles  — 

Project  Strategy  and  Support 
Overall  Plan 

Planned  sizes  and  rates  used  to  compute  times 

Quality  Preparation 

Planned  Defects  Injected/Removed 

Planned  quality  indicator  values  are  acceptable  — 

Balanced  Plan 

Project  Risk  Analysis 

Launch  Report  Preparation 

Management  Review 

Launch  Postmortem  — 

Approved  for  Public  Release 
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BE 


TxP  Working  Activities 


Logging  time 
Logging  defects  ■ 

Tracking  EV  ■ 

Using  PROBE  in  Planning  phase 

Entering  actual  sizes  in  Postmortem  phase 

Defining  Defect  Types 

Using  Review  checklists 

Holding  periodic  team  meetings  — — 

Following  an  agenda  during  team  meetings 

Performing/reporting  on  assigned  roles 

Reviewing  action  items 

Reviewing  assigned  goals  and  risks 

Maintaining  project  plan  and  workbook 
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TxP  Analyzing  Activities 


Evaluate  plan  vs.  actual  schedule  hours 
Evaluate  plan  vs.  actual  component  hours 
Evaluate  plan  vs.  actual  component  sizes 
Evaluate  team  performance  vs.  goals  and  quality  plan 
Evaluate  plan  vs.  actual  quality  of  components 
Update  planning  data  for  schedule  hours  — 


Update  planning  data  for  lifecycle  time-in-phase  %s 
Update  planning  data  for  productivity  rates 
Update  planning  data  for  defect  densities 


Update  planning  data  for  defect  rates  and  yields 
Update  planning  data  for  quality  indicator  thresholds 
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Transitions 


Training  & 
First  Launch 


3-part  TPI 
Training 

^Process 

Modeling 

v^First  Launch 


Product  Size 
Definition 


^Define  size 
measures 

S Add  Planning 
and  Postmortem 
phases 

^Begin  use  of 
PROBE 


Defect 

Removal 


^Define  Defect 
Types 

^Refine  Processes 
with  Defect 
Removal  Phases 


Quality 

Indicators 


^Define  Product 
Quality  Indicators 

^Define  Process 
Quality  Indicators 


Stages 


Planning 

Activities 


Working 

Activities 


Analyzing 

Activities 
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Time-Based  Postmortem 


•  The  team's  most  consistent  data  at  first  is  time 

-  Time  on  Task  by  Team  Member 

-  Planned  vs.  Actual  Time  by  Component 

-  Planned  vs.  Actual  Time  by  Product/Service  Type 

-  Planned  vs.  Actual  Time  by  Workflow 


•  Sample  Time  Log 


Logged  To 

Start  Time 

Delta 

/Project/PRT  FY2014/Common/Monthlies/Dct  201 3/TPI  Coaching/AV-8B  SW/Do 

Tue  Oct  OS  08:00:52  PDT2013 

0:10 

/Project/PRT  FY20 14/TP  l/C  heckpoints/H-1  SIT- Aug  2013  part  2/Follow-up 

Tue  Oct  08  08:20:49  PDT2013 

0:17 

/Project/PRT  FY2Q  14/TP  l/C  heckpoints/H-1  SIT- Aug  2013  part  2/Follow-up 

Tue  Oct  08  03:45:47  PDT2013 

0:05 

/Project/PRT  FY2Q  14/TP  l/C  heckpoints/H-1  SIT- Aug  2013  part  2/Follow-up 

Tue  Oct  08  09:08:18  PDT2013 

0:21 

/Project/PRT  FY2Q14/TPI/Checkpoints/H-1  SIT- Aug  2013  part  2/Follow-up 

Tue  Oct  03  10:00:13  PDT2013 

0:05 

/Project/PRT  FY2Q14/Common/Monthlies/Oct2013/TPI  Coaching/Informal/Do 

Tue  Oct  08  10:17:40  PDT2013 

0:10 

/Project/PRT  FY2Q  14/TP  l/C  heckpoints/H-1  SIT- Aug  2013  part  2/Follow-up 

Tue  Oct  08  10:29:44  PDT2013 

0:10 

/Project/PRT  FY2  01 4/TP  l/C  heckpoints/H-1  SIT- Aug  2013  part  2/Follow-up 

Tue  Oct  08  10:54:50  PDT2013 

0:07 

/Project/PRT  FY2  01 4/TP  l/C  heckpoints/H-1  SIT- Aug  2013  part  2/Follow-up 

Tue  Oct  08  11:23:09  PDT2013 

0:12 

/Project/PRT  FY2Q14/Common/Monthlies/Oct2013/TPI  Coaching/AV-3B  SW/Do 

Tue  Oct  03  13:53:34  PDT2013 

0:07 

/Project/PRT  FY2Q14/Common/Monthlies/Oct2013/TPI  Coaching/CEI  SSAT/Do 

Tue  Oct  08  14:01:02  PDT2013 

2:16 

/Project/PRT  FY2014/Common/Monthlies/Dct2013/TPI  Coaching/CEI  SSAT/Do 

Tue  Oct  08  16:59:59  PDT2013 

0:33 

/Project/PRT  FY201 4/Comm on/Monthlies/Oct 20 13/TP  1  Coaching/CEI  SSAT/Do 

Tue  Oct  08  17:44:18  PDT2013 

0:42 

/Project/PRT  FY201 4/Comm  on/Monthlies/Oct  20 13/TP  1  Coaching/H-1  SIT/Do 

Wed  Oct  09  06:35:14  PDT2013 

0:11 

/Project/PRT  FY2014/Common/Monthlies/Oct  201 3/TPI  Coaching/H-1  SIT/Do 

Wed  Oct  09  07:20:14  PDT2013 

0:21 

/Project/PRT  FY2Q14/Comm on/Monthlies/Oct  201 3/TPI  Coaching/CCS/Do 

Wed  Oct  09  03:00:06  PDT2013 

0:48 

nav^Tai  r 
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ime  on  Task  by  Team  Member 


Planned  vs.  Actual  Time  by  Week 


Avg 

Planned 

Actual 

%  of 

Time 

Count 
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7.7 

110% 
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10.8 

120% 

10 

1 

6.7 

67% 

11 
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12 

21 

13.5 

113% 

13 

0 

14 

0 

15 

0 

16 

0 

17 

0 

18 

0 

19 

0 

20 

0 

21 

1 

15.2 

72% 

22 

0 

%  of  Time  Entries  Ending  in 


30% 

25% 

20% 

15% 

10% 

5% 

0% 


1234567890 


%  of  entries  across  all  numbers  70% 
extra  on  5s  and  0s  30% 


•  Time  Log  analysis 

-  Accuracy  &  precision  of  estimates 

-  Real-time  logging  vs.  backfilling 
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Time  by  Component 


By  Component 


0 ) 

E 


fU 
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4—? 
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50  100  150 

Planned  Time  (hrsl 


By  Component  Type 


*  t 


f  f  J*  Jtfj? 

^  ^  ^  V'0  ^  V'0  v~° 


Planned  Time  ■  Actual  time 


Study  any  points  in  red  regions 
Adjust  team  productivity  rates  for  next  cycle 
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Time  by  Workflow 


Plan  Time 

Actual  Time 

Plan  % 

Actual  % 

Act  -  Plan 

Next  Plan 

Normalized 

Planning 

68:06 

41:03 

2.4% 

1.4% 

-1.1% 

2.4°A 

2.3% 

High-Level  Design 

248:01 

251:46 

8.8% 

8.4% 

-0.4% 

8.8°4 

8.4% 

HLD  Inspection 

103:07 

65:44 

3.7% 

2.2% 

-1.5% 

3.7°A 

3.5% 

Detailed  Design 

356:52 

339:32 

12.7% 

11.3% 

-1.4% 

12.7 

12.2% 

Detailed  Design  Review 

129:06 

90:59 

4.6% 

3.0% 

-1.6% 

4.6°4 

4.4% 

Test  Development 

61:44 

34:58 

2.2% 

1.2% 

-1.0% 

2.2°A 

2.1% 

Detailed  Design  Inspection 

294:44 

220:51 

10.5% 

7.4% 

-3.1% 

7A°A 

7.1% 

Code 

435:48 

575:10 

15.5% 

19.2% 

3.7% 

19.2°4 

18.4% 

Code  Review 

143:08 

112:39 

5.1% 

3.8% 

-1.3% 

5.1°4 

4.9% 

Compile 

21:04 

16:06 

0.7% 

0.5% 

-0.2% 

0.7°A 

0.7% 

Unit  Test 

349:58 

485:12 

12.4% 

16.2% 

3.7% 

16.2°4 

15.5% 

Code  Inspection 

365:50 

444:37 

13.0% 

14.8% 

1.8% 

13.0°4 

12.5% 

Build  and  Integration  Test 

189:47 

290:05 

6.7% 

9.7% 

2.9% 

6.7°A 

6.5% 

Postmortem 

46:48 

29:26 

1.7% 

1.0% 

-0.7% 

1.7  % 

1.6% 

Total 

2814:03 

2998:08 

100.0% 

100.0% 

1 04.3Q4 

100.0% 

•  Isolate  times  for  one  kind  of  activity 

-  Analyze  &  discuss  big  differences - 


-  Proposed  planned  %s  for  next  cycle 
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Size-Based  Postmortem 


•  Once  the  team  has  consistent  size  data... 

-  Productivity  Rates  by  Team  Member 

-  Planned  vs.  Actual  Size  by  Component 

•  Example  of  Size  Documentation 


BASE  PROGRAM  SIZE  1 

Estimated  |  |  Actual  | 

SIZE  SIZE 

BASE  SIZE  (B) 

DELETED  SIZE  (D) 

MODIFIED  SIZE  (M) 

150  150 

75  75 

8  8 

BASE  ADDITIONS  1 

Estimated  |  |  Actual  | 

ITEMS  REL.  SIZE  SIZE  SIZE  ITEMS 

Report  Altitude 

add  more  rows  for  base  additions... 

-  -  10  13 

Total:  |10  fl3 

PARTS  ADDITIONS  ' 

Estimated  |  |  Actual  | 

ITEMS  REL.  SIZE  SEE  UR  SIZE  ITEMS  UR 

GUI 

I/O  ▼  1  Medium  ▼  16  O  22  O 

Altimeter  Referencing 

add  more  rows  for  parts  additions... 

Logic  ▼  1  Large  ▼  23  O  33  n 

Total:  39  55 
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roductivity  Rates  by  Team  Member 


•  All  individuals  have  their  own  rates. ..per  product  type 


H  PROBE 


J 


Method 
Correlate 
with 


PROBE  Method  C3  for  Time 


New  &  Changed  LOC 


Time 


Estimate 


%  range 


70 


L.  Regression 

Projection  =  ???? 
BetaO  =  -43.9625 
Betal  =  1.6923 
r2  =  0.7169 
p  =  3.35% 

Variance  =  38413.02 
StdDev  =  195.992 
Range  =  ???? 

DPI  =  7777 
l_pi  =  7777 


Average 

Projection  =  ???? 
BetaO  =  0.0 
Betal  =  1.5869 


Filter- 


Chart... 


Close 


Betal  (mimites/LOC)  =  1,587 
PSP  Prod  uctivity  Rate  ( LOCs/Hr)  =  38 
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Size  by  Component 


Previous  Cycle  Components 

Time  Diff 


t(A-P)/P] 


Current  Cycle  Components 

Time  Diff 
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Quality-Based  Postmortem 


•  Getting  a  handle  on  defects  usually  happens  last 

-  Defect  Injection  Rate  by  Phase 

-  Defect  Measures  by  Defect  Type 

-  Defects  Injected  by  Phase 

-  Defects  Removed  by  Phase 


•  Sample  Defect  Log 


j:  Project 

j:  /Non  Project/PSP  for  Engineers/Program  5 

ID 

1 

Type 

Environment 

Injected 

Test 

Removed 

Test 

Time 

0.6 

Count 

1 

Description 

did  not  configure  wires  properly  during  board  test 

Date 

09/03/2009 

|i  /Non  Project/dev/PMPT/JDAM  Cross  Range 

1 

Interface 

Design 

Design  Review 

5.3 

1 

forgot  to  consider  general  architucture  classes  in  my ... 

00/25/2010 

/Non  Project/dev/PMPT/JDAM  Cross  Range 

2 

Assignment 

Design 

Design  Review 

10.3 

15 

didn't  initialize  parameters 

09/10/2010 

j:  /Non  Project/dev/PMPT/JDAM  Cross  Range 

3 

Documentation 

Design 

Design  Review 

5 

1 

did  not  draw  data  flow  arrows  in  correct  direction  bet... 

10/13/2010 

ji  /Non  Project/dev/PMPT/JDAM  Cross  Range 

4 

Interface 

Design 

Design  Review 

0.6 

1 

forgot  to  flesh  out  paras  fortune  Compute  Angle 

11/17/2010 

|i  /Non  Project/dev/PMPT/JDAM  Cross  Range 

5 

Interface 

Design 

Design  Review 

1.2 

1 

forgot  to  flesh  out  paras  fortune  Compute  Range 

11/10/2010 

j!  /Non  Project/sw  history/P rod  A  Reqt  3 

1 

Function 

Code 

Test 

11 

1 

inverted  to  variables 

01/20/2009 

j:  /Non  Project/sw  history/Prod  A  Reqt  3 

2 

Unclear 

Design 

Design  Review 

0.9 

3 

did  not  give  vars  initial  values... 

10/11/2011 
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Defect  Injection  Rate  by  Phase 
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12000 


efect  Measures  by  Defect  Type 


Defect  Fix  Times 


I.i.i.i.i 

<?  ^  ^ 


f  *  <f  S 


<y 


Sorted  by  Fix  Time 


How  many  doesn't 
always  matter 


Defect  Counts 


I 


T -  - 1 -  - 1 -  - 1 -  - 1 -  - 1 -  - 1 -  - T 


I  , 


Z5  *  ^ 


^  J  Jr  4 
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Defects  Injected/ Removed  by  Phase 


Planned  vs.  Actual  Phase  Yields 


100.0% 


80.0% 


60.0% 


40.0% 


20.0% 


0.0% 


DLD  Review  DLD  Code  Review 
Inspection 


□  Planned  Phase  > 


Height  of  Red  above 
Green  indicates  how 
many  mistakes  were  in 
the  product  at  that 
phase  of  development 
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200 


Cumulative  Defects  Injected  and  Removed 


<200  remaining 


>600  remaining 
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iCum  Actual  Injected  aCum  Removed  (70%)  a  Cum  Removed  (50%)  aCum  Actual  Removed 


Slide  33 


Approved  for  Public  Release 


nav^Tai  r 


TxP  Postmortem 


•  Only  after  the  team  knows  what  level  of  process 
performance  results  in  a  quality  product,  then  they 
can  set  goals  and  compare  planned  values  to  actual 
values. 


RATIOS 

Plan 

Actual 

Phase 

0.36 

0.27 

DLD  Review/DLD  Ratio 

0.82 

0.59 

DLD/Code  ratio 

0.33 

0.20 

Code  Review/Code 

0.00 

3.15 

Compile  Defect  Density 

8.86 

7.81 

Unit  Test  Defect  Density 

REVIEW  RATES  (LOCs/hr) 

Plan 

Actual 

Phase 

336 

829 

DLD  review 

71 

136 

DLD  inspection 

147 

266 

CODE  review 

60 

62 

CODE  inspection 

Cost  of  Quality  (COQ) 

Topic 

Plan 

Actual 

%  Appraisal  COQ 

36.8% 

30.8% 

%  Failure  COQ 

19.9% 

26.1% 

Appraisal  /  Failure  Ratio  (AFR) 

1.85 

1.18 
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Things  to  Remember 


•  As  a  team's  process  evolves  from  TPI  to  TxP,  the 
analysis  of  their  data  needs  to  evolve  too 

•  Focus  on  what  is  value-added  to  the  team  and  they 
will  strive  to  collect  the  data 

•  This  analysis  gives  them  insight  into  the  quality  of 
their  processes  used  to  produce  their  products  and 
provide  their  services 
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NAVAIR  Process  Resource  Team 

Brad  Hodgins 


bradley.hodgins@navy.mil 
(760)  939-0666 
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Trademarks  and  Service  Marks 


•The  following  are  service  marks  of  Carnegie  Mellon  University. 

-  Team  Software  ProcessSM 

-  TSPSM 

-  Personal  Software  ProcessSM 

-  PSPSM 


•The  following  are  registered  trademarks  of  Carnegie  Mellon 
University. 

-  Capability  Maturity  Model® 

-  CMM® 

-  Capability  Maturity  Model  Integration® 

-  CMMI® 

-  CERT® 
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Acronym  List 


•A/FR  -  Appraisal  Failure  Ratio 

•CMM  -  Capability  Maturity  Model 

•CMMI  -  Capability  Maturity  Model  Integration 

•COQ  -  Cost  of  Quality 

•DLD  -  Detailed-Level  Design 

•EV  -  Earned  Value 

•HLD  -  High-Level  Design 

•LOC  -  Line  of  Code 

•NAVAIR  -  Naval  Air  Systems  Command 

•PI  -  Process  Improvement 

•PROBE  -  PROxy-Based  Estimating 

•PRT  -  Process  Resource  Team 

•PSP  -  Personal  Software  Process 

•SEI  -  Software  Engineering  Institute 

•TSP  -  Team  Software  Process 

•TPI  -  Team  Process  Integration 

•TxP  -  Team  [topic  name]  Process 
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